Read the vocabulary sets through Lexicon._sets (#754, 4i-a) - #989

Sub-phase 4i-a of #754's Lexicon surface, as decided in #988. This is internal plumbing only: the parse reads the vocabulary sets through one internal object instead of Lexicon's public fields, and the three tools that read a lexicon's sets move to the same object. Nothing public changes and no parse moves.

What changed

• _lexicon._LexiconSets is a frozen, slotted object with one frozenset per _VOCAB_FIELDS entry:
  • the 20 fields that 4i-b makes internal;
  • surnames and given_names, which stay public. They are included because they sit in the same roster loops (_vocab.in_any_wordlist, claimed_as_non_name, _assemble._listed_whole), so each loop reads one object.
• The _sets slot. Each Lexicon builds a _LexiconSets in __post_init__, from its normalized fields. It lives in a slot _sets inherited from a plain base class (_HoldsSets):
  • A dataclass field, even one built init=False, would change dataclasses.fields(Lexicon), the pickled state and the 2.8 pickles' layout check.
  • The slot is not compared, hashed or pickled. A load, dataclasses.replace and | rebuild it.
• The readers moved. Every library module but _lexicon (the pipeline stages, _render, _parser) now reads lexicon._sets.titles where it read lexicon.titles. So do tools/name_usage.py, tools/es_given_names.py and tools/differential/local_agreement.py. None of the tools moved to the index:
  • local_agreement reads a pack parser's lexicon, which the shipped index does not describe.
  • name_usage reads suffix_acronyms_ambiguous, a marker the compile derives and no index row states.
• Reads moved, re-counted with the 2026-11-01 recipe:
  • The recipe's 111 library reads in 12 files break down as:
    • 8 belong to _lexicon itself and stay;
    • 2 are TailRead.titles reads, not reads of a lexicon;
    • 101 moved.
  • Also moved: 5 reads of surnames and given_names, and 5 roster getattrs.
  • In the tools, 16 of the 17 reads moved. The 17th is local_corpora's counter, which is not a lexicon.
• Reads: lines. They still name a set as Lexicon.<field>. AGENTS.md now says the stages read it through _sets, and the release checklist's grep becomes \b(policy|lexicon|lex)(\._sets)?\.[a-z][a-z_]*. It keeps printing the field name and no longer prints a bare _sets. A note says that _classify's blanket Reads: line covers twenty fields: the 17 the grep lists in the module, and the rest read through _vocab.
• Field order. _LexiconSets is built by keyword. tests/v2/test_lexicon.py holds its field names to _VOCAB_FIELDS, in order, and holds each attribute to be the field's own set.

The guard

tests/v2/test_layering.py walks every library module but _lexicon, plus every file under tools/ (40 files), and refuses a vocabulary field read off a lexicon. It runs one test per _VOCAB_FIELDS entry.

• In the library:
  • A read passes only if it goes through <x>._sets, or through a name sets that every binding in the module assigns from a _sets.
  • sets bound any other way fails: an annotated assignment, a walrus, unpacking, a for target, a parameter or a with target.
  • TailRead's read.titles is named as not a lexicon.
  • A getattr fails if its object is a lexicon or an alias of one, or if it names a field as a constant off anything but a _sets.
• In the tools, whose counters reuse the field names, a read fails if it is taken off lexicon, lex, <x>.lexicon, Lexicon.default(), or a name some binding gives one of those (an assignment, a walrus, a Lexicon annotation). This includes the AGENTS.md recipe's L = Parser().lexicon.
• Not followed, and stated in the check's docstring:
  • a lexicon handed through a container, a call's return, or an attribute other than .lexicon;
  • aliases per scope rather than per module.
• Negative controls:
  • A synthetic read in each of 9 spellings (5 library, 4 tool) must be refused, for every field.
  • Each binding form of sets and of an alias has its own row, in the library and in the tools.
  • _UNGUARDED_READS records what the check refuses on today's sources with ._sets stripped out:
    • 114 library reads: 106 attribute loads, 5 roster getattrs (3 of them through the sets that stripping makes a lexicon) and 3 sets = bindings;
    • 16 tool reads.
• Mutation check: on the built tree, changing one lex._sets.titles_ambiguous in _classify back to lex.titles_ambiguous failed exactly [titles_ambiguous].

Frames

Measured on Python 3.11.17 with the benchmark's counter (50 names, four-digit counter), b55d4d33 against this branch:

There is no new frame per token and no new frame per parse. Wall clock over 6,000 names stayed within the run-to-run spread (96 to 98 µs per name on both trees). _CALL_BASELINE is unchanged.

How 4i-b plugs in

4i-b changes how _sets is filled, and nothing that reads it:

• Filling. _sets is built from the lexicon's readings, compiled by compile_readings, instead of from its public fields.
• The default stays lazy. The default's _sets comes from nameparser/config/_compiled.py, as _default_lexicon reads it today, so parse("John Smith") still loads no index.
• Derived markers. The markers the views will read (a listed suffix word's kind, the patronymic particles) join _LexiconSets as attributes. _vocab.suffix_kind and classify's particle kind then read them there instead of the index.
• The slot. Since dataclasses.fields(Lexicon) changes in 4i-b anyway, _sets may become a private field then, and the base class can go.
• _parser._apply_choice reads _sets but still writes the public fields through dataclasses.replace. 4i-b has to give it the readings to write until 4i-c retires it.
• What 4i-b also breaks, outside what the guard holds:
  • the tests' public-field reads;
  • AGENTS.md's recipe for the titles in no suffix set, L.titles - L.suffix_acronyms - L.suffix_words on L = Parser().lexicon. 4i-b rewrites it over readings_of or _sets.

The same plan is recorded in the new decisions/vocabulary-model entry (2026-11-02 #754, sub-phase 4i-a), and the RFC's 4i-a row is marked landed.

A design-docs review re-measured every number in the decisions entry and reproduced them. It raised three low findings, all fixed before the first push: a roster wording, the AGENTS.md scope sentence, and a missing tool-side lex control row.

An independent review of the PR found no blocking defect. Its five small items are fixed in a second commit:

• the keyword build and the field-order test;
• the widened guard;
• the tuned grep;
• AGENTS.md's "Adding a curated sub-set" paragraph, which said _VOCAB_FIELDS is read by two rosters. There are three, _assemble._listed_whole among them, and _LISTED_FIELDS leaves four fields out;
• the test count below.

Verification

• uv run pytest: 34,457 passed, 29 skipped, without namedivider. The run includes tests/v2/test_snapshot.py, unchanged, and test_cldr_agreement.py, no move.
• uv run pytest -n0 -m serial: 42 passed.
• uv run mypy, uv run ruff check and tools/vocab/build.py --check are clean.
• The suite also passes on Python 3.15, and without namedivider-python installed.
• No public change: dataclasses.fields(Lexicon) is unchanged (24 fields), reprs are untouched, and tests/v2/data/lexicon_v28.pickle still loads.
• Layering is unchanged; ALLOWED was not extended.

🤖 Generated with Claude Code